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SUMMARY 


In the application of the method of least squares to data reduction, it may 
he desired to compute the effect of a single additional observation on the regres- 
sion coefficients and their covariance matrix obtained previously. Simple exact 
formulas by which the new observation may be included are derived in this report 
for the case where the observation errors are uncorrelated. The starting point 
for the derivation is the matrix formulation of the well-known least-squares 
solution, and succeeding steps in the derivation do not require any further 
application of statistical theory. The resulting recursion expressions are iden- 
tical to those developed from statistical filter theory for the case of uncor- 
related errors. 


INTRODUCTION 


In the application of the method of least squares to data reduction, it 
would be desirable to be able to calculate the effect of an additional observa- 
tion on a previous estimate of the (so-called) regression coefficients without 
resorting to the lengthy process of matrix inversion. Also, If one or more of 
the observations which have entered into the solution should appear, upon later 
examination, to be "wild, " a simple procedure for removing these observations 
would be useful. Such procedures would permit rapid calculations of the type 
needed for smoothing and extrapolating time histories, as is done in the proc- 
essing of radar measurements, for example. 

This report describes how simple recursion formulas for both adding and 
subtracting observations may be derived from the well-known least-squares normal 
equations by simple applications of matrix algebra. In the derivation, the 
unknown quantities to be estimated are assumed to be constants. It is possible, 
however, to extend the formulas to the problem of estimating the state variables 
of a simple form of dynamic system. The resulting formulas are then the same as 
those derived by the application of optimal control theory to the problem of 
estimation and filtering in reference 1. References 2 and 3 present different 
derivations of the recursion formulas and applications of them to space naviga- 
tion. Reference 4 is an early treatment of the application of least squares to 
data reduction. 


SYMBOLS 


A* 


A 

X 


*n 

x n+l 

Y* 


a 




known coefficients in regression equation 
coefficients of weighted regression equation 
true regression coefficients 

original least-squares estimate of regression coefficients 
revised estimate of regression coefficients 

observed quantities from which x n was determined 
weighted observations 
additional observation 
weighted value of y* 

known coefficients corresponding to added observation 

weighted values of a* 

true errors in observed quantities Y 

true error in y* 

probable error in y* 

original covariance matrix 



DO 

V 

t 


revised covariance matrix 
difference between jc^ and jc n +}7j 
identity matrix 

weight for added observation equation 
time 
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Matrix symbols: 


O 

LJ 

II II 

□ 

LI 


column matrix 


row matrix 


rectangular matrix 


square matrix 


diagonal matrix 


Prime denotes transposition of a matrix. ^Example: The notation 

denotes the column matrix formed by transposing the row matrix L*J- 



DERIVATION OF METHOD 


The derivation of the method of this report begins with a brief statement of 
the least -squares solution. Succeeding steps In the derivation are elementary 
matrix operations. 


Let the system of simultaneous equations relating observations 



to 


the unknowns 



be 


{r»> - |a*|{x} + {€ Y *} (l) 

The number of observations n is greater than the number of elements of , 

and the errors ey* are assumed to be random with a Gaussian distribution, zero 
mean, variance cr y , and not correlated. Under these conditions, the probable 

value of obtained from n equations and denoted by ^ x n } > is obtained by 

weighting each observation equation by multiplying it by the reciprocal of the 
variance Oy of the corresponding observation, and then solving for coefficients 

/x n ) which will minimize the sum of the squares of the weighted residuals. If 
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the weighted observations are denoted by {y} and the weighted observation 
coefficients by ||a|, the probable value of is (see ref. 5 ) 


{*„} - [a'aTVKy} (2) 

The matrix [a'a] 1 in equation (2) is the covariance matrix of {^n)> • A 
proof of this statement appears in the appendix. For convenience, it will be 
denoted by j^C^j . Statistically, the covariance matrix is the expected value of 

the product formed by postmultiplying the difference between ^x^ and its 
expected value by the transpose of the difference. The diagonal elements of the 
covariance matrix [pnj are "the squares of the probable errors in the corre- 
sponding elements of {^n}>- Because of its statistical significance, it is 
generally desirable to compute the covariance matrix, even though the solution 

for ^x n A> could be performed without actual matrix inversion (by Crout's method, 
for example) . 

It will be assumed that a set of observation equations has been solved, as 

in equation (2), to obtain ^n} and- the covariance matrix jjc^j . Suppose that 
an additional observation is then made and it is desired to compute the effect of 
this observation on the probable value of and on the covariance matrix. 

The additional observation y* is related to by the equation 


y* “ L a *J { X } + e y* (3) 

The weighted equation corresponding to equation ( 3 ) is: 

y = [_aj {x} + (4) 

°y 


Revising the Estimate of 



Without Matrix Inversion 


The least-squares solution for the revised estimate 



may be written: 
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(5) 


{ x n+l} = 


A’ j a’ 


_A_ 

a 


-1. 


A' | a’ 


where partitioning has been used to show the added elements. The matrix 

is the revised covariance matrix Jc n +ij • This matrix may 


|a' ! a' I 

I 

| 

a II 

1 1 ! 

* 


be written 

as 


[? n +3 " 


or 


A' A + 


[Cn+3 “ |L C n] -1 + { a ')L a J| 

Equation ( 5 ) may then be written 

{ x n + l} = 0 n ] _1 + {a’} M ( Ml ( Y ) + {»'}*! 


{ a '} L a J 


-1 


-1 


(6) 


(7) 


-l 


( 8 ) 


An expression for ^n+l^ i n ^ erms ^n} ma y written as follows. 
Equation ( 8 ) may be written as 

[On]' 1 * (a'} Laj] {x„.l} - |*'| {*} ♦ (a'} 7 (9) 

If equation ( 9 ) is multiplied through by Jjc^j , there is obtained the equation 

Li] - [On] {a'}L a J_ (vi) ■ [ C »] l A ’l ( Y ) + &G ( a ') y (l0) 

By equation (2), 

[On] ll A ' II ( Y > - {*»} < U > 

For convenience^ let = * Then equation (10) may he written 


lil + { k '} L a J|{ x n+l} = { x n} + ( k '} y 


(12) 
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From equation (12), 


{ x n+l} = L 1 ! + ( k '} L a J {>} + L 1 ! + { k '}L a J { k '} y to) 

By making use of the identity, 

Lfl - M u] [l 1 ! - {*'} w] - [fl <*> 

simple expressions for the terms 

n _1 r n -1 

L 1 ! + { k ')L a J { x n} 811(1 Jjl + ( k, }L a JJ ( k '} 

in equation (13) may t>e obtained. Expansion of equation (l4) gives: 

lfl + {“'} L»J - Lfl + M L*J MW - lfl <«) 

Postmultiplication of equation (15) by {k'J> gives: 

\f\ + { k ’}L a J ( k ') + l/l + { k ’}L a J { k '}L a J{ k ’} = { k '} (l6) 

The product |^aj <^k f ^ is a scalar quantity. Let this quantity he denoted 
by B. Then equation (l6) may be written 

r n -i 

|_l] + {k'}|_a| {k'} (1 + B) = {k'} (17) 

Therefore, 

J/l + { k ’} L a J ( k ') - rTB^') (l8) 

Postmultiplication of equation (15) by {*nf gives 

[|jl + { k '}L a J ( x n) + L 1 ! + ( k '}L a J { k '}L a J { x n} s { x n} d9) 

By equation (l8), equation (19) may be written 

Ifl + ML?! {>} + rri{ k, }L a J{ x n} " ^ (20> 
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from which 


Lfl + {k’} l_aj {xn} - {x n } - 3 -^ { k '} L a J { Xn } 

By equations (l8) and ( 21 ), equation (13) may now be written as 

{>1} - {%} - + rr^( k '> y 


{>l} - {%} + j^TS {*’}(* ‘ L a J {*-} 


Equation (23) furnishes the means for revising the estimate of . However, 

the matrix {k 1 } and the scalar B are derived from the covariance matrix of 
the previous estimate. Therefore, if another observation is to be added, the 
revised covariance matrix [c n +ij must also be computed. This computation can 
also be done without matrix inversion. 


Evaluating £c n +£J 

Let [*] be the difference between [c n +£J and [0^] so that 

[Cn+l] - [C^ + gc] 


By definition of the inverse of a matrix. 


[Cn + l] _1 [Cn] + [Cn+l]" 1 ^] = \f] 


From equation (7) it is seen that 


[C n +3 _1 = En ]" 1 + ( a '}L a J 


Therefore, equation (25) may be written 


lil + ( a ’} L a JC Cn ] + + { a ’> L a J[Ml = l/l 
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or 


{ a '} L a J[ C *] + in]" 1 ^] + {a'} |_ a J&3 = [°] 


(28) 


Multiplication by [ n ] and solving for [ac] gives: 

B = " Ifl + [pn] {a’} |_aj [c n ] {a'} |_a) fc n ] 

But |Cj 7 ] [a'] = {k'y by previous definition. Also, because 

metrical, [_aj [Ci] = [kj. Thus, equation (29) may be written: 

(Ml - -[bn - M L a J {*'} L k J 

By use of equation (l8), equation (30) becomes 

- - rr^{ k '}L k J 


(29) 

C n | is sym- 


(30) 


(3D 


The expression for calculating [pn+l] without matrix inversion may therefore 
be written 


(C n +l] - [Cn] - j-nK k ’} W 


(32) 


In order to keep the weighting process flexible, the final equations may be 
written in terms of the unweighted observation equation and a weighting factor 
w by making the following substitutions in equations (23) and (32): 



[_kj = w \a*J [ Cn ] 


B = w 2 \a*] [ Cn ] {a* '} 


Now let 

id fed - L k *J 
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and 

L?*J &,]{>'} - B* 

Equation (32) may then he written 

1 + w^B' 




( 33 ) 


Equation (23) becomes 

{*n + l} * {*n} + ~ {( k *>'}(y* - a * { X "}) 


(3*1) 


When the weighting factor is chosen to be the reciprocal of the probable error 
of the observation ay*, equations (33) (3^) become 




(35) 


and 


( x n + l) * {%} ♦ - W W) 


(36) 


Recursion Formulas for Deleting an Observation 

These formulas are derived by as suming that 0n+f] < (^ x n+lj > are 

known and solving for [Cj^j and . The observation to be deleted is y, 

and its corresponding observation coefficients are [_aj- From equation (7)> it 
may be seen that 

,-l 


n-i 


C n ] = fn+g" " { a ’}L a J 


(37) 


where L a J is now the row of coefficients in the observation equation to be 
deleted. 


By equation (2), 


{xn} - lin.1]- 1 - {a’}W[ H© 


(38) 
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or 


[Cn+l]" 1 - 0'}L a J { x n} = I A '||{y} (39) 

In order to obtain an expression involving ^x n+1 ^ , the matrix ^a'J-y is 
added to and subtracted from the right-hand side of equation (39), and the 
resulting expression is multiplied through by £c n+ £J . Thus, equation ( 39 ) 
becomes 

L 1 ! - [c n +l] ( a '} L a J { x n} = pn+l] jj| A ’ || { Y } + (a’} 7 - {a 1 } yj (40) 

By equations ( 7 ) and (8), 



Therefore, equation (40) may he written 

L 1 ! - £n+l) { a '}L a J { x n} = { x n+l} - |f n +I) {&'} y (42) 

Let [pn+lj = {^1*} * Then equation (42) becomes 

Lfl + { k l’} L a J { x n} = { x n+l} - { k l’}y (43) 

Equation (43) is similar in form to equation (12). By the same procedure used 
in simplifying equation (12), it may be shown that equation (43) can be written 
as 

{xn} • {x n+1 } + 3 ( L a J - y) (U) 

where Bp = |^aj ^kp 1 . 

By following procedures similar to those expressed in equations (24) to (32), 
it may be shown that 

[ c nJ - [C n+1 ] - Bl { k l’} L k lJ ^5) 
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Application to Dynamic Systems 

When the quantities to be estimated are variables which describe the instan- 
taneous state of a dynamic system, and when the observed quantities are not 
occurring simultaneously, the method of least squares can be applied if the state 
variables at one time can be expressed in terms of those at another time in the 
following way: 


{ x t,2> = [*(*2* { X t,l} {k6) 

where the transition matrix [^(^25 ls ^- e P en< i en ^ on ly on the times tg 

and tq at which states and occur 0X1(1 ma y be calculated 

from a knowledge of the dynamics of the system. Under these conditions, the 
state at the time of any particular observation may be expressed in terms of the 
state at the time of the last observation. The necessary simultaneous equations, 
in the form of equation (l), can then be set up to obtain a least-squares estimate 
of the state at the time tjj of the nth observation. Then, considering the fact 

that at time t n+ q 


- [»(W t-nJK^n} (V7) 

it is easily shown that the least-squares estimate ^xt^ri+ 1 ^ ts calculated by 
substituting [*(t n+1 j t n )] { x t,n} for { x n} 911(1 substituting 
[®(t n+1 j t n J] [Cn] p'(t n+1 ; t n J] for in equations ( 36 ) and ( 37 ) and in the 

definitions of and B*. Thus, 

{x^n+l} = (*(^+ 1 ' b n J] {*t,n} 

+ {**}( y* - l_a*J [*M b n)] { x t,n}) 

Oy + B 

where 

= (j>(t n+ i; b nT) [pn] If* tn 3 “C a '3“ 

and 

B* = [_a^J [^("tn+lJ b n)3 [pn] (® '(^n+li t nJ]^ a *'} 
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The covariance matrix of 



is 


[c n+ l] = [*(W t n )][c n ][t'(t n+1 ; t n )] - 1 - (k»')b*J 

(Ty + B 


( 49 ) 


DISCUSSION 


Uses of Recursion Formulas 

In any experimental procedure where data are accumulated over a period of 
time and where the least-squares method is to be used, the data reduction may be 
started as soon as enough observations have been made to give a solution. The 
remaining observations may be accounted for as they are made. Each time the 
recursion formulas are applied, the revised solution forms a basis for adding a 
new observation. The solution may be checked for wild points by calculating the 
residuals of the observations which have entered the solution. If any observa- 
tion has an excessively large residual, it may be deleted and replaced by another 
observation if desired. 

In certain data-smoothing procedures, a small section of, for example, a 
time history of experimental data Is fitted by a polynomial which would not fit 
the entire time history. As the data are accumulated, data points are added to 
one end of the small section and deleted from the other end. The recursion 
formulas In this report should be very useful In procedures of this sort. 


Effects of Correlated Errors 

In this report it has been assumed that the observation errors were uncor- 
related. That Is, a knowledge of the error in any one observation gives no 
Information as to the probable error In any other observation. 

When the observation errors are known to be correlated, different recursion 
formulas may be used if the correlation coefficients are known. The recursion 
formulas In references 2 and 3 take into account the possibility of correlated 
errors. In many practical cases, either the observation errors are not correlated 
or their correlation coefficients are not known. In these cases the ordinary 
weighted least- squares estimate is the best which can be obtained. Reference 6 
treats the subject of correlated errors from the standpoint of least squares. 


Significance of Covariance Matrix 

It is obvious from equation (2) that the covariance matrix of the estimated 
coefficients does not contain any information obtained from the observations 
directly. The variances of the observed quantities, which enter as weighting 
factors, are presumed to be known. Consequently, the covariance matrix which 
results from the estimation of the unknown coefficients by any given set of 
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observations can be computed before any observations are made. Likewise, by 
means of the recursion expression, the effect of a single observation on the 
covariance matrix can be predicted without actually making the observation. If 
in some experimental program there is a choice of observations which may be added, 
the optimum observation may be chosen by precomputing the effect of each observa- 
tion and choosing the one which gives the greatest reduction in the covariance 
matrix. 


Langley Research Center, 

National Aeronautics and Space Administration, 

Langley Station, Hampton, Va., October l 8 , 1962. 
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APPENDIX 


PROOF THAT [c^j IS COVARIANCE MATRIX OF {x n J> 


If the weighting factors which were applied to equation (l) are arranged in 


a diagonal matrix denoted by 
equation (2) may be written 


} it may be seen that 




| A* = A jj and 


w = 


a* t 


Cy 


A* 


-1 




(Al) 


From the definition of a covariance matrix and under the condition previ- 
ously assumed that the errors in the observations are uncorrelated, it 

follows that the covariance matrix of {y*> is the diagonal matrix • Let 


this matrix be denoted by 


I. 


{ x n} - 


- Then equation (Al) may be written as 
1 


r - ^-1 


ri* I 


I. 


Ml 


i 


I. 


-1 


{*•> 


(A2) 


Let 


mi 


L 


{y*} 


(A3) 


For example, in an equation of the form 


{p} - H(r} 

the covariance matrix of is given in terms of the covariance matrix of 

by the following relationship derived from the theory of the propagation of 
error: 


L -H I 


hi 
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is then 


The covariance matrix of T^_ 


r 



" 1 

-i 




-i 

I 

= 

Ik -I 

5 


< 


I Y 

kl 

hj-J 



L 


L tJ 





Because 



is symmetrical, equation (A^) becomes 


(AM 


I 


XJ 


= II A 


* i 


triii. 


-i 


or, by the definition of fCrfJ given in the text 

r_ , ~i -l 
'Y 


I 

lSj 


= A- 


KI " fn 


(A5) 


(A6) 


From equations (A2) and (A3), again making use of the definition of [jCnj , 



(AT) 


Again, by the theory of the propagation of errors, and making use of the fact 
that the transpose of [Cn] is [cQ , the covariance matrix of ^x n ^ , denoted 



Substituting 

becomes 





according to equation (A 6), 


CAB) 


equation (A8) 
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